Welcome to the February 2020 report from the Reproducible Builds project. One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes. The motivation behind the reproducible builds effort is to provide the ability to demonstrate these binaries originated from a particular, trusted, source release: if identical results are generated from a given source in all circumstances, reproducible builds provides the means for multiple third-parties to reach a consensus on whether a build was compromised via distributed checksum validation or some other scheme. In this month s report, we cover:

• Media coverage & upstream news A new paper on reproducible containers, Ruby updates, etc.
• Distribution work More work in Debian, openSUSE & friends.
• Software development Updates and improvements to our tooling.
• Getting in touch How to contribute, more venues for discussion.

If you are interested in contributing to the project, please visit our Contribute page on our website.

Media coverage & upstream news Omar Navarro Leija, a PhD student at the University Of Pennsylvania, published a paper entitled Reproducible Containers that describes in detail the workings of a new user-space container tool called DetTrace:

All computation that occurs inside a DetTrace container is a pure function of the initial filesystem state of the container. Reproducible containers can be used for a variety of purposes, including replication for fault-tolerance, reproducible software builds and reproducible data analytics. We use DetTrace to achieve, in an automatic fashion, reproducibility for 12,130 Debian package builds, containing over 800 million lines of code, as well as bioinformatics and machine learning workflows.

There was also considerable discussion on our mailing list regarding this research and a presentation based on the paper will occur at the ASPLOS 2020 conference between March 16th 20th in Lausanne, Switzerland. The many virtues of Reproducible Builds were touted as benefits for software compliance in a talk at FOSDEM 2020, debating whether the Careful Inventory of Licensing Bill of Materials Have Impact of FOSS License Compliance which pitted Jeff McAffer and Carol Smith against Bradley Kuhn and Max Sills. (~47 minutes in). Nobuyoshi Nakada updated the canonical implementation of the Ruby programming language a change such that filesystem globs (ie. calls to list the contents of filesystem directories) will henceforth be sorted in ascending order. Without this change, the underlying nondeterministic ordering of the filesystem is exposed to the language which often results in an unreproducible build. Vagrant Cascadian reported on our mailing list regarding a quick reproducible test for the GNU Guix distribution, which resulted in 81.9% of packages registering as reproducible in his installation:

$ guix challenge --verbose --diff=diffoscope ...
2,463 store items were analyzed:
  - 2,016 (81.9%) were identical
  - 37 (1.5%) differed
  - 410 (16.6%) were inconclusive

Jeremiah Orians announced on our mailing list the release of a number of tools related to cross-compilation such as M2-Planet and mescc-tools-seed. This project attemps a full bootstrap of a cross-platform compiler for the C programming language (written in C itself) from hex, the ultimate goal being able to demonstrate fully-bootstrapped compiler from hex to the GCC GNU Compiler Collection. This has many implications in and around Ken Thompson s Trusting Trust attack outlined in Thompson s 1983 Turing Award Lecture. Twitter user @TheYoctoJester posted an executive summary of reproducible builds in the Yocto Project: Finally, Reddit user tofflos posted to the /r/Java subreddit asking about how to achieve reproducible builds with Maven and Chris Lamb noticed that the Linux kernel documentation about reproducible builds of it is available on the kernel.org homepages in an attractive HTML format.

Distribution work

Debian Chris Lamb created a merge request for the core debian-installer package to allow all arguments and options from sources.list files (such as [check-valid-until=no] , etc.) in order that we can test the reproducibility of the installer images on the Reproducible Builds own testing infrastructure. (#13) Thorsten Glaser followed-up to a bug filed against the dpkg-source component that was originally filed in late 2015 that claims that the build tool does not respect permissions when unpacking tarballs if the umask is set to 0002. Matthew Garrett posted to the debian-devel mailing list on the topic of Producing verifiable initramfs images as part of a wider conversation on being able to trust the entire software stack on our computers. 59 reviews of Debian packages were added, 30 were updated and 42 were removed this month adding to our knowledge about identified issues. Many issue types were noticed and categorised by Chris Lamb, including:

• openstack_pkg_tools_python_shebang_and_dependencies
• build_path_in_code_generated_by_dgbus_codegen
• random_argument_handling_in_javatools_jh_build
• fortran_captures_build_path
• etc.

openSUSE In openSUSE, Bernhard M. Wiedemann published his monthly Reproducible Builds status update as well as provided the following patches:

• python-rpm-macros (do not save time-based .pyc files for tests)
• solfege (filesystem ordering issue sent upstream via email; package is orphaned upstream)
• DVDStyler (zip timestamps, submitted upstream)
• python-pikepdf (recreate unreproducible .pyc files)

Software development

diffoscope diffoscope is our in-depth and content-aware diff-like utility that can locate and diagnose reproducibility issues. It is run countless times a day on our testing infrastructure and is essential for identifying fixes and causes of nondeterministic behaviour. Chris Lamb made the following changes this month, including uploading version 137 to Debian:

• The sng image utility appears to return with an exit code of 1 if there are even minor errors in the file. (#950806)
• Also extract classes2.dex, classes3.dex from .apk files extracted by apktool. (#88)
• No need to use str.format if we are just returning the string. [ ]
• Add generalised support for ignoring returncodes [ ] and move special-casing of returncodes in zip to use Command.VALID_RETURNCODES. [ ]

Other tools disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues. This month, Vagrant Cascadian updated the Vcs-Git to specify the debian packaging branch. [ ] reprotest is our end-user tool to build same source code twice in widely differing environments and then checks the binaries produced by each build for any differences. This month, versions 0.7.13 and 0.7.14 were uploaded to Debian unstable by Holger Levsen after Vagrant Cascadian added support for GNU Guix [ ].

Project documentation & website There was more work performed on our documentation and website this month. Bernhard M. Wiedemann added a Java Gradle Build Tool snippet to the SOURCE_DATE_EPOCH documentation [ ] and normalised various terms to unreproducible [ ]. Chris Lamb added a Meson.build example [ ] and improved the documentation for the CMake [ ] to the SOURCE_DATE_EPOCH documentation, replaced anyone can with anyone may as, well, not everyone has the resources, skills, time or funding to actually do what it refers to [ ] and improved the pre-processing for our report generation [ ][ ][ ][ ] etc. In addition, Holger Levsen updated our news page to improve the list of reports [ ], added an explicit mention of the weekly news time span [ ] and reverted sorting of news entries to have latest on top [ ] and Mattia Rizzolo added Codethink as a non-fiscal sponsor [ ] and lastly Tianon Gravi added a Docker Images link underneath the Debian project on our Projects page [ ].

Upstream patches The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann (for the openSUSE distribution):
  • rdiff-backup fixed, build date in Python manpage
  • click-man fixed, toolchain, build date in Python manpage
  • rpm report filesystem-order related nondeterminism
  • python-enaml report nondeterministic .enamlc compiler output
  • k9s date
  • python-xcffib fix a test with a race condition
• Chris Lamb:
  • #943956 re-opened against snakemake.
  • #950630 filed against designate.
  • #950936 filed against pynwb.
  • #950942 filed against python-oslo.reports.
  • #951357 filed against mate-desktop (forwarded upstream).
  • #951573 filed against javatools.
  • #952493 filed against xavs2.
  • #952694 filed against snapd-glib (forwarded upstream).
  • #952762 filed against openstack-pkg-tools.
  • ccextractor issues with MacOSX BSD date not understanding GNU date options
  • python-django (Always build the documentation in English)
  • python_example (sort glob / readdir)

Vagrant Cascadian submitted patches via the Debian bug tracking system targeting the packages the Civil Infrastructure Platform has identified via the CIP and CIP build depends package sets:

• #950410 filed against autogen.
• #950417 & 950416 & 950415 filed against autoconf.
• #950419 filed against m4.
• #950444 filed against intltool.
• #950585 filed against binutils-dev.
• #950603 filed against yodl-doc.
• #950606 filed against gdb-source.
• #950704 filed against automake-1.15.
• #951031 filed against libtool.

Testing framework We operate a fully-featured and comprehensive Jenkins-based testing framework that powers tests.reproducible-builds.org. This month, the following changes were made by Holger Levsen:

• Debian:
  • Configured an instance of David Bremner s builtin-pho database of .buildinfo files. This has resulted in so that we now know that Debian bullseye contains 4,557 source packages for the amd64 architecture without corresponding .buildinfo files and 25,668 source packages with .buildinfo files. [ ][ ][ ][ ][ ]
  • Forward mails addressed to buildinfo@ to the root user. [ ]
  • Temporarily revert using backports [ ][ ] and use devscripts from buster-backports [ ].
  • Update URL for PureOS package set. [ ]
  • Deprioritise the scheduling of older old packages in bullseye and unstable. [ ][ ]
  • Treat the bullseye distribution like unstable when scheduling the i386 architecture, to allow the latter to catch up a bit. [ ]
• Misc/other:
  • Disable the last active Alpine builder too for now. [ ]
  • Improve generated HTML output. [ ][ ]
  • Ignore Fedora jobs. [ ]
  • Include links to failed and unstable jobs in HTML output. [ ]
  • Start weighting job importance and five a lot more weight to nodes acting as a proxy for others. [ ][ ][ ]
  • Add new job to calculate the overall system health for tests.reproducible-builds.org for usage with Jelle van der Waa s Reproducible Builds status display. [ ]

In addition, Mattia Rizzolo added an Apache web server redirect for buildinfos.debian.net [ ] and reverted the reshuffling of arm64 architecture builders [ ]. The usual build node maintenance was performed by Holger Levsen, Mattia Rizzolo [ ][ ] and Vagrant Cascadian.

Getting in touch If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.
• Twitter: @ReproBuilds
• Reddit: /r/ReproducibleBuilds
• Mailing list: rb-general@lists.reproducible-builds.org

---

This month s report was written by Bernhard M. Wiedemann, Chris Lamb and Holger Levsen. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.

Here's what happened in the Reproducible Builds effort between Sunday November 5 and Saturday November 11 2017: Upcoming events On November 17th Chris Lamb will present at Open Compliance Summit, Yokohama, Japan on how reproducible builds ensures the long-term sustainability of technology infrastructure. We plan to hold an assembly at 34C3 - hope to see you there! LEDE CI tests Thanks to the work of lynxis, Mattia and h01ger, we're now testing all LEDE packages in our setup. This is our first result for the ar71xx target: "502 (100.0%) out of 502 built images and 4932 (94.8%) out of 5200 built packages were reproducible in our test setup." - see below for details how this was achieved. Bootstrapping and Diverse Double Compilation As a follow-up of a discussion on bootstrapping compilers we had on the Berlin summit, Bernhard and Ximin worked on a Proof of Concept for Diverse Double Compilation of tinycc (aka tcc). Ximin Luo did a successful diverse-double compilation of tinycc git HEAD using gcc-7.2.0, clang-4.0.1, icc-18.0.0 and pgcc-17.10-0 (pgcc needs to triple-compile it). More variations are planned for the future, with the eventual aim to reproduce the same binaries cross-distro, and extend it to test GCC itself. Packages reviewed and fixed, and bugs filed Patches filed upstream:

• Bernhard M. Wiedemann:
  • clang - ASLR affects objective-C binaries.
• Chris Lamb:
  • nbsphinx (merged) - Random UUIDs used as element selectors.
  • stardicter (merged) - SOURCE_DATE_EPOCH support.
  • stetl - Build path in documentation.

Patches filed in Debian:

• Bernhard M. Wiedemann:
  • #881231 filed against chasen - Uninitialized memory from struct padding written into data files.
• Adrian Bunk:
  • #881453 filed against primesieve - FTBFS.
• Chris Lamb:
  • #881089 filed against stardicter - (merged) SOURCE_DATE_EPOCH.
  • #881094 filed against nbsphinx - random UUIDs.
  • #881157 filed against designate - build path.
  • #881217 filed against python-stetl - build path.
  • #881258 filed against sphinx-intl - drop date.
  • #881259 filed against soundmodem - build path.
  • #881262 filed against node-module-deps - build path.
  • #881474 filed against phatch - random memory address.
• Daniel Kahn Gillmor:
  • #881152 filed against npth - build path.

Patches filed in OpenSUSE:

• Bernhard M. Wiedemann:
  • i4l-base (merged) - Uninitialized memory written to output.

Reviews of unreproducible packages 73 package reviews have been added, 88 have been updated and 40 have been removed in this week, adding to our knowledge about identified issues. 4 issue types have been updated:

• Add randomness_in_files_generated_by_pinyin_gen_binary_files.
• Add build_path_captured_in_assembly_objects.
• Add timestamps_in_ifo_files_generated_by_python_stardicter.
• Update timestamps_in_source_generated_by_rcc.

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (69)
• Andreas Beckmann (3)
• Dmitry Shachnev (1)
• Graham Inggs (1)

diffoscope development Mattia Rizzolo uploaded version 88~bpo9+1 to stretch-backports. reprotest development

• Ximin Luo:
  • build: add comment that util-linux confirmed bug in nsenter, awaiting fix.
  • Make --print-sudoers work for --env-build as well.

reproducible-website development

• Holger Levsen:
  • rws3: add OTF as sponsor
  • rws3: add F-Droid, riot-os.org
• Chris Lamb:
  • Move the "contribute" page from the Debian wiki to /contribute/ on our main website.
• Eitan Adler:
  • Fix typo in FreeBSD mailing list.

theunreproduciblepackage development

• Bernhard M. Wiedemann:
  • aslr: document per-process workaround
  • aslr: add examples

tests.reproducible-builds.org in detail

• Mattia Rizzolo:
  • reproducible archlinux: enable debugging mode
  • reproducible archlinux: don't use hidden files for the package lists
  • reproducible fedora: don't use hidden files for the package lists
  • udd-query: move from public-udd-mirror.xvm.mit.edu to udd-mirror.debian.net
  • udd-query: remove the temporary file with a trap in case this script is called with the wrong argument, and in case of failures, etc, the temporary file would be left around otherwise
  • reproducible debian: schroot-create: drop the reproducible gpg keyring into /etc/apt/trusted.gpg.d/ instead of using apt-key add
  • reproducible debian: setup_pbuilder: drop the reproducible gpg keyring into /etc/apt/trusted.gpg.d/ instead of using apt-key add
  • reprodocible debian: setup_pbuilder: stop installing gnupg2 in our chroot, not needed anymore now
  • Mattia also merged and deployed some commits from others this week.
• Alexander 'lynxis' Couzens
  • reproducible_lede: use correct place/variable to save results: Results on remote nodes are expected to be under $TMPDIR, which defined by openwrt_build. RESULTSDIR is undefined on the remote node
  • reproducible_lede: enable building all packages again, after it was disabled to improve the debug speed.
  • reproducible_lede: correct given path for node_cleanup_tmpdirs & node_save_logs- reproducible_lede: enable CONFIG_BUILDBOT to reduce inodes while building.
• kpcyrd:
  • reproducible-archlinux: try porting abs to asp
  • reproducible-archlinux: explicitly sync packages
  • reproducible-archlinux: use sudo for pacman
• Hans-Christoph Steiner:
  • reproducible fdroid: point jenkins to canonical URL
  • reproducible_fdroid: separate testsuite into its own job
  • reproducible fdroid: sync upstream script names with jenkins.debian.net, make things self-documenting by reusing the same names everywhere.
  • reproducible_fdroid_test: make script executable
• Chris Lamb:
  • Move some IRC announcements to #debian-reproducible-changes.
• Holger Levsen:
  • reproducible LEDE: try to deal gracefully with problems and report
  • as usual, Holger merged many of the above commits and deployed them.

Misc. This week's edition was written by Ximin Luo, Bernhard M. Wiedemann, Chris Lamb and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday October 29 and Saturday November 4 2017: Past events

• From October 31st November 2nd we held the 3rd Reproducible Builds summit in Berlin, Germany. A full, in-depth report will be posted in the next week or so.

Upcoming events

• On November 8th Jonathan Bustillos Osornio (jathan) will present at CubaConf Havana.
• On November 17th Chris Lamb will present at Open Compliance Summit, Yokohama, Japan on how reproducible builds ensures the long-term sustainability of technology infrastructure.

Reproducible work in other projects

• Alan Pope linked to the status of "asset recording" in SnapCraft, a concept related to .buildinfo files.

Packages reviewed and fixed, and bugs filed

• Chris Lamb:
  • #880714 filed against cappuccino.
  • #880803 filed against python-kafka.
  • #880804 filed against debci.
  • #880827 filed against roundcube.
• Steve Langasek:
  • #880550 filed against g2.
• Juro:
  • curl (SOURCE_DATE_EPOCH)
  • OpenSSL (SOURCE_DATE_EPOCH)
• Bernhard M. Wiedemann:
  • openSUSE/ant (use SOURCE_DATE_EPOCH)
  • openSUSE/mariadb (drop INFO_BIN)
  • openSUSE/libsmbios (drop embedded build log)

Reviews of unreproducible packages 7 package reviews have been added, 43 have been updated and 47 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (44)
• Andreas Moog (1)
• Lucas Nussbaum (7)
• Steve Langasek (1)

Documentation updates

• Ulrike Uhlig:
  • Add documentation about system images.
• Holger Levsen:
  • List all speakers of the DebConf17
  • Add news about the Berlin event
• Chris Lamb:
  • Add recent talks to resources.html.
  • Clarify link to full definition.
  • Cachebust CSS files.
• Bernhard M. Wiedemann:
  • Fix openSUSE website link

diffoscope development Version 88 was uploaded to unstable by Mattia Rizzolo. It included contributions (already covered by posts of the previous weeks) from:

• Mattia Rizzolo
  • tests/comparators/dtb: compatibility with version 1.4.5. (Closes: #880279)
• Chris Lamb
  • comparators:
    • binwalk: improve names in output of "internal" members. #877525
    • Omit misleading "any of" prefix when only complaining about one module in ImportError messages.
  • Don't crash on malformed "md5sums" files. (Closes: #877473)
  • tests/comparators:
    • ps: ps2ascii > 9.21 now varies on timezone, so skip this test for now.
    • dtby: only parse the version number, not any "-dirty" suffix.
  • debian/watch: Use HTTPS URI.
• Ximin Luo
  • comparators:
    • utils/file: Diff container metadata centrally. This fixes a last remaining bug in fuzzy-matching across containers. (Closes: #797759)
    • Fix all the affected comparators after the above change.
• Holger Levsen
  • Bump Standards-Version to 4.1.1, no changes needed.

strip-nondeterminism development Version 0.040-1 was uploaded to unstable by Mattia Rizzolo. It included contributions already covered by posts of the previous weeks, as well as new ones from:

• Mattia Rizzolo:
  • Declare that strip-nondeterminism doesn't need root to build
  • Add my key to debian/upstream/signing-key.asc m disorderfs development

---

Version 0.5.2-2 was uploaded to unstable by Holger Levsen. It included contributions already covered by posts of the previous weeks, as well as new ones from:

• Chris Lamb:
  • Add a reproducible builds example, highlighting the non-intuitive recommendation to sort instead of shuffle.
  • Add myself to AUTHORS.
  • debian/watch: Use HTTPS URI.
• Holger Levsen:
  • debian/control: use /git/ instead of /cgit/ URL for Vcs-Browser
  • debian/compat: Bump to compat level 10 & raise build-depends to debhelper >= 10.2.5~.
  • debian/control: Bump Standards-Version to 4.1.1.

reprotest development

• Ximin Luo:
  • Mention certificate expiry errors in documentation as a potential issue with faketime

buildinfo.debian.net development

• Chris Lamb:
  • Support UID filtering
  • Add initial "by-hash" API endpoint.

tests.reproducible-builds.org

• Mattia Rizzolo:
  • archlinux: enable schroot building on pb4 as well
  • archlinux: don't install the deprecated abs tool
  • archlinux: try to re-enable one schroot creation job
• lynxis
  • lede: replace TMPDIR -> RESULTSDIR
  • lede: openwrt_get_banner(): use locals instead of globals
  • lede: add newline to $CONFIG
  • lede: show git log -1 in jenkins log
• Holger Levsen:
  • lede: add very simple landing page
• Juliana Oliveira Rodrigues
  • archlinux: adds pacman-git dependencies
• kpcyrd
  • archlinux: disable signature verification when running in the future
  • archlinux: use pacman-git until the next release
  • archlinux: make pacman fail less early
  • archlinux: use sudo to prepare chroot
  • archlinux: remove -rf for regular file
  • archlinux: avoid possible TOCTOU issue
  • archlinux: Try to fix tar extraction
  • archlinux: fix sha1sums parsing

Misc. This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Mattia Rizzolo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday October 22 and Saturday October 28 2017: Past Events

• On Tuesday 24th October, Chris Lamb presented at All Things Open 2017 in Raleigh, NC, USA.
• On Wednesday 25th October, Holger Levsen presented at the Open Source Summit Europe in Prague, Czech Republic.
• On Saturday 28th October, Chris Lamb presented at freenode.live in Bristol, UK.

Upcoming/current events

• From October 31st November 2nd we will be holding the 3rd Reproducible Builds summit in Berlin, Germany.
• On November 8th Jonathan Bustillos Osornio (jathan) will present at CubaConf Havana.

Documentation updates Bernhard Wiedemann started The Unreproducible Package which "is meant as a practical way to demonstrate the various ways that software can break reproducible builds using just low level primitives without requiring external existing programs that implement these primitives themselves. It is structured so that one subdirectory demonstrates one class of issues in some variants observed in the wild." Reproducible work in other projects Hush, a fork of ZCash, opened an issue into reproducible builds. A new tag was added to lintian (lint checker for Debian packages) to ensure that changelog entry timestamps are strictly increasing. This avoids certain real-world issues with identical timestamps, documented in Debian #843773. Packages reviewed and fixed, and bugs filed Patches sent upstream:

• Bernhard M. Wiedemann:
  • gtranslator, embedded build timestamps
  • libgda, embedded build timestamps
  • mariadb, embedded build timestamps
  • nim, embedded build timestamps

Debian bug reports:

• Chris Lamb:
  • #879569 filed against argagg.
  • #879913 filed against sdlgfx.
  • #879914 filed against fmtlib.

Reviews of unreproducible packages 14 package reviews have been added, 35 have been updated and 28 have been removed in this week, adding to our knowledge about identified issues. 1 issue type has been updated:

• Add nondeterministic_ordering_in_documentation_generated_by_doxygen.

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (4)

strip-nondeterminism development Version 0.040-1 was uploaded to unstable by Mattia Rizzolo. It included contributions already covered by posts of the previous weeks, as well as new ones from:

• Mattia Rizzolo:
  • png.pm: Don't open the original file in write mode

reprotest development Development continued in git:

• Ximin Luo:
  • New features:
    • Support a domain_host variation.
    • Support a --print-sudoers feature.
  • Documentation:
    • Note some caveats about the existing git versions as a self-reminder not to release it yet.
    • Updates about our assumptions and rearrange sudo into its own section.
  • Bug fixes:
    • main: When dropping privs, make sure the user can still move in theroot.
    • tests: fix, need to preserve env for su
    • build: Don't fail when the build produces a broken symlink
    • main, presets: Properly drop privs when running the build. (Closes: #877813)
  • Code quality:
    • Improve logging to try to get to the bottom of the jenkins failures
    • Tweak tests to avoid some build dependencies
    • build: Name temporary directories after reprotest not autopkgtest

buildinfo.debian.net development Development continued in git:

• Chris Lamb:
  • New features:
    • Add API endpoint to fetch specific .buildinfo files for a certain package/version/architecture, and optimise it. (Closes: #25)
  • Bug fixes:
    • Always show SHA256, regardless of viewport size. (Closes: #27)
    • Actually filter by source package

reproducible-website development

• Holger Levsen:
  • RWS3 Berlin 2017:
    • Add CoyIM, Arch Linux, LEDE, LEAP, subuser.org, Bazel, coreboot.
    • Make some sponsors visible.
    • Add short paragraph explaining that registration is mandatory.

Misc. This week's edition was written by Ximin Luo, Chris Lamb, Bernhard M. Wiedemann and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Debian subversion

• Continued work on updating the packaging
  • Thanks to some hints from Daniel Shahaf, realized the entire build was running from the binary target, thus as root. This was the cause for the test failures I had been seeing last month.
• Pushed some long standing patches upstream
  • r1811786

vim

• Uploaded version 2:8.0.1226-1
  • Improved syntax highlighting for debian/control files. Thanks to Mattia Rizzolo for reminding me to look at deb-src-control(5).
  • Updated debsources and debchangelog syntax highlighting for new/unsupported releases in Debian and Ubuntu. Josh Triplett spotted a syntax error which will be fixed in the next upload.
• Reported upstream that a particular test was failing on many of our buildds.
  • Upstream released a fix to sleep longer before moving on in the test.
  • I proposed that it may be better to check for the actual desired state instead.

---

libvterm

• Proposed an API for reporting focus events, which was merged. (r713)

---

pangoterm

• Proposed a change to use libvterm's new API, which was merged. (r598)

---

vim-gnupg

• Fixed an issue that prevented saving the encrypted buffer if the file was opened with a relative path and the user had subsequently changed the working directory. (vim-gnupg#81)

---

neovim

• Added tests and merged my PR to ignore leading modifier commands (e.g., :silent, :keeppatterns) when using 'inccommand'. (neovim#6967)
• Potentially root-caused a Neovim hang as being caused by a Debian-specific patch to libuv.

---

vim

• During a discussion in a Neovim issue, I was reminded that :lockmarks doesn't affect Vim's '[ and '] marks. I had previously worked around this in vim-gnupg, but seeing someone else run into the issue spurred me to look into it more. Unlike many other marks, Vim doesn't distinctly track these "last change/yank" marks. They're tracked the same way that the range of the current "operation" are -- the b_op_start and b_op_end items in the buf_T struct. This makes the problem a little more touchy, since those are changed in many places. After I had a working prototype, I decided to take a step back and see if there were tests related to what I was going to be changing. Unfortunately, there weren't many and none that dealt with autocmds. These marks are especially relevant for Vim's autocmds because many of those provide the plugin author with information through the '[ and '] marks. That means :lockmarks needs to preserve the state of the marks for the user, but still communicate the relevant information for the autocmds. To that end, I submitted a pull request to test how these marks are affected by autocmds. Once that's merged, I'll feel more comfortable with proposing the existing prototype, which only covers :write, :read, the :diff* commands, and filtering.

Here's what happened in the Reproducible Builds effort between Sunday October 15 and Saturday October 21 2017:

• The Tails project published a report on how they made their ISO images reproducible.
• dpkg 1.19.0 was uploaded, including support for:
  • Ordering the "unused substitution" warnings to prevent superfluous differences between logs of package builds on the Reproducible Builds test framework. (#870221)
  • A new Build-Kernel-Version field in .buildinfo files that can be generated with a new dpkg-genbuildinfo --always-include-kernel option. (#873937)

Past events

• On Saturday 21st October, Holger Levsen presented at All Systems Go! in Berlin, Germany on reproducible builds. A video recording is available already.

Upcoming events

• On Tuesday 24th October, Chris Lamb will present at All Things Open 2017 in Raleigh, NC, USA.
• On Wednesday 25th October, Holger Levsen will present at the Open Source Summit Europe in Prague, Czech Republic.
• On Saturday 28th October, Chris Lamb will present at freenode.live in Bristol, UK.
• From October 31st November 2nd we will be holding the 3rd Reproducible Builds summit in Berlin, Germany. If you are working in the field of reproducible builds, you should definitely attend. Please see our public invitation mail and contact us if you have any questions.

New York University sessions A three week session will be held at New York University to work on reproducibilty issues in conjunction with the reproducible builds community. Students from the Application Security course will be working for two weeks to work on the reproducible builds effort.

• On Tuesday 24th Oct Ed Maste from FreeBSD will be presenting some reproducible builds work for students.
• On From Tuesday 24th of October to Monday 7th of November students will work on fixing reproducibility issues brought up by the community. A milestone presentation will be held by Santiago Torres-Arias and Preston Moore.
• On Tuesday 7th November Holger Levsen will join the NYU team to wrap up the work.

Packages reviewed and fixed, and bugs filed

• Adrian Bunk:
  • #878883 filed against stoken.
  • #878896 filed against liquidsoap.
  • #878980 filed against belenios.
  • #879042 filed against tuxpaint-config.
  • #879223 filed against websockify.
• Chris Lamb:
  • #878707 filed against python-amqp (randomness in documentation).
  • #878708 filed against geographiclib (build paths).
  • #879169 filed against live-build (timestamps).
• Gilles Filippini:
  • #878818 filed against ovito.

The following reproducible builds-related NMUs were accepted:

• Chris Lamb:
  • gerstensaft (#777414)
  • miniupnpd (#860266)
  • rest2web (#833438)

Patches sent upstream:

• Bernhard M. Wiedemann:
  • singularity (tar+gz)
  • redis (date)
  • tbb (date)
  • nested (sort)
  • nested (date)

Reviews of unreproducible packages 41 package reviews have been added, 119 have been updated and 54 have been removed in this week, adding to our knowledge about identified issues. 2 issue types were removed as they were fixed:

• too_much_input_for_diff
• docbook_to_man_one_byte_delta

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Aaron M. Ucko (1)
• Adrian Bunk (49)
• Anthony DeRobertis (1)
• Daniel Schepler (1)
• Gilles Filippini (1)
• James Cowgill (1)
• Matthias Klose (1)
• Matthias Klumpp (1)
• Nobuhiro Iwamatsu (1)

diffoscope development

• Ximin Luo:
  • Tentative support for for parallel diffs

strip-nondeterminism development Version 0.039-1 was uploaded to unstable by Chris Lamb. It included contributions already covered by posts of the previous weeks, including:

• Chris Lamb:
  • Clojure considers the .class file to be stale if it shares the same timestamp of the .clj. We thus adjust the timestamps of the .clj to always be younger. (#877418)
  • dh_strip_nondeterminism: Log which handler processed a file. (#876140)
  • bin/strip-nondeterminism: Print a warning in --verbose mode if no canonical time specified.
  • Use HTTPS URI in debian/watch.

reprotest development

• Ximin Luo:
  • Hopefully fix the autopkgtest tests
• Santiago Torres-Arias:
  • An Arch Linux User Repository package for reprotest has been created.]

tests.reproducible-builds.org

• Holger Levsen:
  • Install rustc on Jenkins for the reproducible-html-build-path-prefix-map-spec job.
• Mattia Rizzolo:
  • health_check: Include the running kernel version when reporting multiple kernel installed in /boot.

Website updates

• Holger Levsen:
  • Berlin 2017:
    • add Qubes OS to participating projects
    • Add NetBSD and EdgeBSD to participating projects
    • Add repeatr.io to participating projects
    • Add disclaimer why only so few projects are listed
    • Add link to Google maps and name the 2nd next subway station on a different line
    • Add first 4 participating projects

Misc. This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen, Santiago Torres & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday October 1 and Saturday October 7 2017: Media coverage

• Bernhard sent another report about the status of Reproducible openSUSE. They currently they are at 478 unreproducible and 11,111 reproducible packages out of 11,821, so also at 93%!
• Holger attempted to get a Reproducible Builds devroom at FOSDEM 2018 but sadly this proposal was not accepted.

Documentation updates

• Christoph Berg created a wiki page about Openjade generated timestamps from DSSSL stylesheets.

Packages reviewed and fixed, and bugs filed

• Bernhard M. Wiedemann:
  • LiE uninitialized memory (need to find upstream)
  • chrony date (merged)
• Chris Lamb:
  • #877375 filed against polygen.
  • #877381 filed against plr.
  • #877384 filed against rcs.
  • #877928 filed against cadvisor.
• jathan:
  • #877470 filed against bsh.

Reviews of unreproducible packages 32 package reviews have been added, 46 have been updated and 62 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (27)

diffoscope development

• Chris Lamb:
  • Don't crash on malformed md5sums files. (Closes: #877473)
  • Improve names in output of "internal" binwalk members. (Closes: #877525)
• Mattia Rizzolo:
  • Fix test compatibility with dtb version 1.4.5

strip-nondeterminism development Rob Browning noticed that strip-nondeterminism was causing serious performance regressions in the Clojure programming language within Debian. After some discussion, Chris Lamb also posted a query to debian-devel in case there were any other programming languages that might be suffering from the same problem.

• Chris Lamb:
  • jar.pm: Clojure considers the .class file to be stale if it shares the same timestamp of the .clj. We thus adjust the timestamps of the .clj to always be younger.. (Closes: #877418)
  • jar.pm, zip.pm: Allow $options member_normalizer callback to support specifying the timestamp.
  • zip.pm: Ensure that we don't try and write an old timestamp; Archive::Zip will do this anyway, just noisily.
  • zip.pm: Calculate the target canonical time in just one place.
  • bin/strip-nondeterminism: Print a warning in --verbose mode if no canonical time specified.
  • jar.pm: Update comment to reflect that NTFS/FAT has a 2s timestamp granularity.
  • jar.pm: s/NTFS/FAT/. Thanks to James Ross.

reprotest development Versions 0.7.1 and 0.7.2 were uploaded to unstable by Ximin Luo:

• New features:
  • Add a --auto-build option to try to determine which specific variations cause unreproducibility.
  • Add a --source-pattern option to restrict copying of source_root, and set this automatically in our presets.
• Usability improvements:
  • Improve error messages in some common scenarios.
    • Fiving a source_root or build_command that doesn't exist
    • Using reprotest with default settings after not installing Recommends
  • Output hashes after a successful --auto-build.
  • Print a warning message if we reproduced successfully but didn't vary everything.
• Fix varying both umask and user_group at the same time.
• Have dpkg-source extract to different build dir if varying the build-path.
• Pass --exclude-directory-metadata to diffoscope(1) by default as this is the majority use-case.
• Various bug fixes to get the basic dsc+schroot example working.

It included contributions already covered by posts of the previous weeks, as well as new ones from:

• Ximin Luo:
  • main: Add a --env-build option for testing different env vars
  • Don't output spurious warnings in tests
  • Add some more notes on the remaining variations
  • Fix the help text for virtual servers

tests.reproducible-builds.org

• Mattia Rizzolo:
  • Re-deploy odxu4a after being reinstalled and renamed from odxu4.
• Vagrant Cascadian:
  • Rename armhf host odxu4 to odxu4a

Misc. This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen, Mattia Rizzolo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday September 24 and Saturday September 30 2017: Development and fixes in key packages Kai Harries did an initial packaging of the Nix package manager for Debian. You can track his progress in #877019. Uploads in Debian:

• Chris Lamb:
  • shadow/1:4.5-1 fixing #857803.
• Holger Levsen:
  • font-uralic/0.0.20040829-6 fixing #854362. (NMU)
  • mpack/1.6-8.2 fixing #777376. (NMU)

Packages reviewed and fixed, and bugs filed Patches sent upstream:

• Bernhard M. Wiedemann:
  • python-numpy build timestamp; merged
  • python-marshmallow build timestamp; merged
  • python-astropy-helpers build timestamp; merged
  • oprofile build timestamp
  • libheimdal build timestamp, hostname, username; upstream exploring alternative fixes
  • openSUSE/Qt hash table seed
  • libkolabxml/xsd ASLR memory location differences; no patch

Reproducible bugs (with patches) filed in Debian:

• Chris Lamb:
  • #877375 filed against polygen.
  • #877381 filed against plr.
  • #877384 filed against rcs.
• Daniel Schepler:
  • #876672 filed against e2fsprogs.
• Vagrant Cascadian:
  • #876657 filed against device-tree-compiler and uploaded 1.4.4-1 with the patch applied.

QA bugs filed in Debian:

• Adrian Bunk:
  • #876641 filed against pcb.
  • #876685 filed against mssh.
  • #876776 filed against fityk.
  • #876845 filed against webkit2-sharp.
  • #876870 filed against apertium-en-es.
  • #877021 filed against breathe.
  • #877031 filed against sextractor.
  • #877054 filed against hypre.
  • #877063 filed against libitpp.
  • #877065 filed against alglib.
  • #877211 filed against ipmiutil.

Reviews of unreproducible packages 103 package reviews have been added, 153 have been updated and 78 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (177)
• Andreas Beckmann (2)
• Daniel Schepler (1)

diffoscope development Mattia Rizzolo uploaded version 87 to stretch-backports.

• Holger Levsen:
  • Bump standards version to 4.1.1, no changes needed.

strip-nondeterminism development

• Holger Levsen:
  • Bump Standards-Version to 4.1.1, no changes needed.

reprotest development

• Ximin Luo:
  • New features:
    • Add a --env-build option for testing different env vars. (In-progress, requires the python-rstr package awaiting entry into Debian.)
    • Add a --source-pattern option to restrict copying of source_root.
  • Usability improvements:
    • Improve error messages in some common scenarios.
    • Output hashes after a successful --auto-build.
    • Print a warning message if we reproduced successfully but didn't vary everything.
    • Update examples in documentation.
  • Have dpkg-source extract to different build dir iff varying the build-path.
  • Pass --debug to diffoscope if verbosity >= 2.
  • Pass --exclude-directory-metadata to diffoscope(1) by default.
  • Much refactoring to support the other work and several minor bug fixes.
• Holger Levsen:
  • Bump standards version to 4.1.1, no changes needed.

tests.reproducible-builds.org

• Holger Levsen:
  • Fix scheduler to not send empty scheduling notifications in the rare cases nothing has been scheduled.
  • Fix colors in 'amount of packages build each day on $ARCH' graphs.

reproducible-website development

• Holger Levsen:
  • Fix up HTML syntax
  • Announce that RWS3 will happen at Betahaus, Berlin

Misc. This week's edition was written by Ximin Luo, Bernhard M. Wiedemann, Holger Levsen and Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday September 17th and Saturday September 23rd 2017: Media coverage

• Christos Zoulas gave a talk entitled Reproducible builds on NetBSD at EuroBSDCon 2017

Reproducible work in other packages

• Paul Eggert reported that TZ=UTC is not a portable setting for the TZ environment variable.

Packages reviewed and fixed, and bugs filed

• Adrian Bunk:
  • #876615 filed against librsvg.
• Bernhard M. Wiedemann:
  • varnish (random IDs)
  • make (sort)
  • nautilus-dropbox (extended date)
  • fontforge (date)
  • votca-csg (merged, date)
  • freeipmi (merged, date)
  • libmypaint (merged, sort)
  • doomsday (merged, sort)
  • asciidoc (help make release with SOURCE_DATE_EPOCH patch)

Reviews of unreproducible packages 1 package reviews was added, 49 have been updated and 54 have been removed in this week, adding to our knowledge about identified issues. One issue type was updated:

• gtk_doc_api_index_full (fixed upstream: #779090)

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (56)
• Bas Couwenberg (1)
• Helmut Grohne (1)
• Nobuhiro Iwamatsu (2)

diffoscope development Version 87 was uploaded to unstable by Mattia Rizzolo. It included contributions from:

• Ximin Luo:
  • comparators/*:
    • Add a test for fallback_recognizes and improve the behaviour
    • Add a fallback_recognizes to work around file(1) #876316. (Closes: #875272)
    • If --force-details then don't skip files with identical md5sums either
    • Add a --force-details flag for debugging
  • presenters.html:
    • Restore the previous more-detailed comment
    • Don't show pointer-cursor when jQuery is disabled
    • Prune all descendants properly (Closes: #875281)
  • difference.py:
    • Also copy self._comment properly, compare self._visuals in equals()
    • In fmap/map_lines, don't forget about self._visuals
    • Use diff_split_lines everywhere
  • tests:
    • Add case for #875281
    • Make test_md5sums less brittle
    • Update test_deb for new string
  • readers: Convert bytes to str in the right place
  • config: Force-set a value if it must be < another and it was not set on purpose (Closes: #875451)
  • Bump minimum Python version to 3.5 as we use syntax introduced by PEP 448
• Chris Lamb:
  • Print a debugging message if we are reading diff from stdin.
  • Compare types with identity not equality.
  • diffoscope.presenters.html: Use logging.py's lazy argument interpolation.
  • debian/control: Bump Standards-Version to 4.1.0.
• Mattia Rizzolo:
  • debian/rules: Place "before" commands before "after" commands.

strip-nondeterminism development

• Chris Lamb:
  • Log which handler procesed a file. (Closes: #876140)
  • Bump Standards-Version to 4.1.0.

reprotest development Version 0.7 was uploaded to unstable by Ximin Luo:

• Ximin Luo:
  • Push use of UNIX return codes to the edges of the program
  • Add a --auto-build option to determine which variations cause unreproducibility
  • Allow umask and user_group to both vary at the same time
  • Generate build names in main instead of build, guard against dupes
  • Pull traceback-printing stuff out of the core code
  • More refactoring, make check() contain only logic that would be changed in an auto-detector
  • Split check() into a coroutine producer and consumer, prepares auto-detection

tests.reproducible-builds.org Vagrant Cascadian and Holger Levsen:

• Re-add and armhf build node that had been disabled due to performance issues, but works linux 4.14-rc1 now! #876212

Holger Levsen:

• Use botch from stretch to fix the jenkins job which create the package sets. (botch is currently uninstallable in sid and from pre-stretch-release times we used a sid schroot to install and use botch.)

Misc. This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday September 10 and Saturday September 16 2017: Upcoming events

• Holger Levsen wrote and published details about our upcoming Berlin summit. Expect a more detailed announced soon and consider planning your travel!

Reproduciblity work in Debian devscripts/2.17.10 was uploaded to unstable, fixing #872514. This adds a script to report on reproducibility status of installed packages written by Chris Lamb. #876055 was opened against Debian Policy to decide the precise requirements we should have on a build's environment variables. Bugs filed:

• Chris Lamb:
  • #875700 filed against gtk+3.0.
  • #875704 filed against gdk-pixbuf.
  • #875792 filed against doit.
• Vagrant Cascadian:
  • #875711 filed against qemu.

Non-maintainer uploads:

• Holger Levsen:
  • fonts-dustin/20030517-11 uploaded, fixing #815723 with patch by Scarlett Clark.

Reproduciblity work in other projects Patches sent upstream:

• Bernhard M. Wiedemann:
  • cadabra2 build timestamp
  • jimtcl build timestamp, fixed another way
  • itpp build timestamp, merged
  • dunst file list ordering, merged
  • HSAIL-Tools hash table ordering, merged
  • kubernetes hash set ordering, no patch

Reviews of unreproducible packages 16 package reviews have been added, 99 have been updated and 92 have been removed in this week, adding to our knowledge about identified issues. 1 issue type has been updated:

• Add build_path_captured_in_assembly_objects

diffoscope development

• Juliana Oliveira Rodrigues:
  • Fix comparisons between different container types not comparing inside files. It was caused by falling back to binary comparison for different file types even for unextracted containers.
  • Add many tests for the fixed behaviour.
  • Other code quality improvements.
• Chris Lamb:
  • Various code quality and style improvements, some of it using Flake8.
• Mattia Rizzolo:
  • Add a check to prevent installation with python < 3.4

reprotest development

• Ximin Luo:
  • Split up the very large __init__.py and remove obsolete earlier code.
  • Extend the syntax for the --variations flag to support parameters to certain variations like user_group, and document examples in README.
  • Add a --vary flag for the new syntax and deprecate --dont-vary.
  • Heavily refactor internals to support > 2 builds.
  • Support >2 builds using a new --extra-build flag.
  • Properly sanitize artifact_pattern to avoid arbitrary shell execution.

trydiffoscope development Version 65 was uploaded to unstable by Chris Lamb including these contributions:

• Chris Lamb:
  • Packaging maintenance updates.
  • Developer documentation updates.

Reproducible websites development

• Holger Levsen:
  • Add a page for the Reproducible Builds World Summit 3 in Berlin 2017.
• Chris Lamb:
  • Moved isdebianreproducibleyet.com to HTTPS.
  • Updated the SSL certificate for buildinfo.debian.net.

tests.reproducible-builds.org

• Vagrant Cascadian and Holger Levsen:
  • Added two armhf boards to the build farm. #874682
• Holger also:
  • use timeout to limit the diffing of the two build logs to 30min, which greatly reduced jenkins load again.

Misc. This week's edition was written by Ximin Luo, Bernhard M. Wiedemann, Chris Lamb, Holger Levsen and Daniel Shahaf & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday September 3 and Saturday September 9 2017: Media coverage

• isdebianreproducibleyet.com was released and subsequently updated.

GSoC and Outreachy updates Debian will participate in this year's Outreachy initiative and the Reproducible Builds is soliciting mentors and students to join this round. For more background please see the following mailing list posts: 1, 2 & 3. Reproduciblity work in Debian

• Chris Lamb filed #874102 filed against texlive-bin to incorporate a proposed upstream change to fix reproducibility issues in generated PDF files.

In addition, the following NMUs were accepted:

• fastforward (#776972) (lamby)
• dtc-xen (#777322) (lamby)
• dhcpping (#777320) (lamby)
• vimoutliner (#776369) (lamby)

Reproduciblity work in other projects

• The Linux kernel announced support for the randstruct GCC plugin.
• "Please make the output of gio-querymodules deterministic" was merged upstream. (lamby)

Patches sent upstream:

• Bernhard M. Wiedemann:
  • gcin: (merged) Uninitialized stack memory
  • html5-parser (merged): Sorting
  • gromacs (merged): Date
  • crawl (merged): Date
  • GCompris-gtk: Date
  • heimdal: Date, hostname
• Chris Lamb:
  • Numpy (#872459) (PR #9652) build path (merged)

Packages reviewed and fixed, and bugs filed

• Adrian Bunk:
  • #874186 filed against svgpp.

Reviews of unreproducible packages 3 package reviews have been added, 2 have been updated and 2 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (15)

diffoscope development Development continued in git, including the following contributions:

• Chris Lamb:
  • Add support for "binwalking" to find (eg.) concatenated CPIO archives. (Closes: #820631)
  • Loosen matching of file(1)'s output to ensure we correctly also match TTF files under file 5.32.
  • Check we identify all CPIO fixtures in tests
  • Make failing a some flake8 tests cause the testsuite to fail. (Currently just "undefined name")
  • Countless style fixups, eg. remove unused imports, removing blank lines from end of flies, etc.
  • Compare None using identity, not equality.
  • diffoscope.diff: Correct reference to self.buf.
  • comparators.utils.file: Correct reference to path_apparent_size.
• Juliana Rodrigues:
  • Sip html_visuals test if 'sng' binary is not available
• Mattia Rizzolo:
  • Numerous PEP8 fixes (eg. E122, E302, E713, etc.)

Mattia Rizzolo also uploaded the version 86 released last week to stretch-backports. reprotest development

• Santiago Torres:
  • Correct string formatting in get_all_servers
• Ximin Luo:
  • Heavy refactoring towards supporting running more than 2 builds
  • Rename append_command to append_to_build_command

tests.reproducible-builds.org

• h01ger:
  • Don't update the stretch package sets anymore
  • Update URL for Tails packages list
  • Disabled the OpenWrt at the request of lynxis as they were broken; if no-one shows up to fix them, we'll probably remove them in the future, as all current development happens within LEDE.
  • Renewed the Let's Encrypt SSL certificates.
• Mattia Rizzolo:
  • Avoid stale temporary files from reproducible_build.sh
  • Avoid removing the temporary directory twice
  • Fix IRC notifications when somebody asks for artifacts
  • Simplify 'if' conditions

Misc. This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen, Mattia Rizzolo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday August 27 and Saturday September 2 2017: Talks and presentations Holger Levsen talked about our progress and our still-far goals at BornHack 2017 (Video). Toolchain development and fixes The Debian FTP archive will now reject changelogs where different entries have the same timestamps. UDD now uses reproducible-tracker.json (~25MB) which ignores our tests for Debian unstable, instead of our full set of results in reproducible.json. Our tests for Debian unstable uses a stricter definition of "reproducible" than what was recently added to Debian policy, and these stricter tests are currently more unreliable. Packages reviewed and fixed, and bugs filed Patches sent upstream:

• Bernhard M. Wiedemann:
  • File ordering:
    • klee-uclibc: sort
    • libdnet: sort
    • libinvm-cim: sort
    • libinvm-cli: sort
  • Embedded build-date timestamps:
    • robinhood: SOURCE_DATE_EPOCH support
    • ceph/rocksdb: SOURCE_DATE_EPOCH support
    • hylafax: use changelog modtime
    • gnucash: use changelog modtime
  • Warzone2100, merged: omit timestamps, sort file lists
• Chris Lamb:
  • glib2.0: sort file lists
  • (old) xorg, merged: SOURCE_DATE_EPOCH support

Debian bugs filed:

• Adrian Bunk:
  • #873608 filed against uhd.
  • #874186 filed against svgpp.
• Chris Lamb:
  • #873625 filed against glib2.0, filed upstream.
  • #874102 filed against texlive-bin.

Debian packages NMU-uploaded:

• Chris Lamb:
  • bittornado/0.3.18-10.3 from #796212
  • cgilib/0.6-1.1 from #776935
  • dict-gazetteer2k/1.0.0-5.4 from #776376
  • dict-moby-thesaurus/1.0-6.4 from #776375
  • dtaus/0.9-1.1 from #777321
  • wily/0.13.41-7.3 from #777360

Reviews of unreproducible packages 25 package reviews have been added, 50 have been updated and 86 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (46)
• Mart n Ferrari (1)
• Steve Langasek (1)

diffoscope development Version 86 was uploaded to unstable by Mattia Rizzolo. It included previous weeks' contributions from:

• Mattia Rizzolo
  • tests/binary: skip a test if the 'distro' module is not available.
  • Some code quality and style improvements.
• Guangyuan Yang
  • tests/iso9660: support both cdrtools' genisoimage's versions of isoinfo.
• Chris Lamb
  • comparators/xml: Use name attribute over path to avoid leaking comparison full path in output.
  • Tidy diffoscope.progress a little.
• Ximin Luo
  • Add a --tool-prefix-binutils CLI flag. Closes: #869868
  • On non-GNU systems, prefer some tools that start with "g". Closes: #871029
  • presenters/html: Don't traverse children whose parents were already limited. Closes: #871413
• Santiago Torres-Arias
  • diffoscope.progress: Support the new fork of python-progressbar. Closes: #873157

reprotest development Development continued in git with contributions from:

• Ximin Luo:
  • Add -v/--verbose which is a bit more popular.
  • Make it possible to omit "auto" when building packages.
  • Refactor how the config file works, in preparation for new features.
  • chown -h for security.

Misc. This week's edition was written by Ximin Luo, Chris Lamb, Bernhard M. Wiedemann and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday August 20 and Saturday August 26 2017: Debian development

• "Packages should build reproducibly" was released in Debian Policy 4.1.0.0. For more background please see last week's post.
• A patch by Chris Lamb to make Dpkg::Substvars warnings output deterministic was merged by Guillem Jover. This helps the Reproducible Builds effort as it removes unnecessary differences in logs of two package builds. (#870221)

Packages reviewed and fixed, and bugs filed Forwarded upstream:

• Debian bug #872728 filed against desktop-file-utils. Filed upstream. (lamby, via a reproducibility issue in Tails).
• Debian bug #872729 filed against gtk+2.0. Filed upstream. (lamby, via a reproducibility issue in Tails).
• sawfish:
  • SOURCE_DATE_EPOCH
  • .tar.gz (merged)
  • Sorting (merged)
• librep:
  • SOURCE_DATE_EPOCH (merged)
  • Sorting (merged)
• zstd (merged) sort
• lirc (solved)
• autogen (tar.gz)

Accepted repoducibility NMUs in Debian:

• Chris Lamb:
  • jsmath-fonts (for bug #792319).
  • xvier (for bug #777330).
• Mattia Rizzolo:
  • aewan (for bug #842864).

Other issues:

• Adrian Bunk:
  • #872675 filed against pdp.
  • #872678 filed against gem.
  • #872860 filed against csound.
  • #872918 filed against sketch.
• Dmitry Shachnev:
  • #872992 filed against pytest-qt.

Reviews of unreproducible packages 16 package reviews have been added, 38 have been updated and 48 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated:

• Add new captures_build_dir_in_qmake_prl_files toolchain issue (Chris Lamb)
• Remove ftbfs_uninvestigated_big_packages (Mattia Rizzolo)

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (37)
• Dmitry Shachnev (1)
• James Cowgill (1)

diffoscope development

• Chris Lamb:
  • Tidy diffoscope.progress a little.
• Santiago Torres-Arias:
  • diffoscope.progress: Support the new fork of python-progressbar. (#873157)

disorderfs development Version 0.5.2-1 was uploaded to unstable by Ximin Luo. It included contributions from:

• Ximin Luo:
  • Add -q,--quiet CLI option
  • Update to latest Standards-Version; no changes required
  • Update debian/changelog
  • Update NEWS
  • Adopt as team maintenance
  • Add my signing key only
  • Add instructions for Debian release
• Holger Levsen:
  • Add myself to uploaders
  • Move package to 'optional' section

reprotest development

• Ximin Luo:
  • Don't vary user_group in the tests - it's not usually possible in automated builders
  • Default verbosity to 0 and silence disorderfs if set
  • Add the ability to vary the user. (#872412)
  • Add TODO

Misc. This week's edition was written in alphabetical order by Bernhard M. Wiedemann, Chris Lamb, Mattia Rizzolo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday August 13 and Saturday August 19 2017: Reproducible Builds finally mandated by Debian Policy "Packages should build reproducibly" was merged into Debian policy! The added text is as follows and has been included into debian-policy 4.1.0.0:

Reproducibility
---------------
Packages should build reproducibly, which for the purposes of this
document [#]_ means that given
- a version of a source package unpacked at a given path;
- a set of versions of installed build dependencies;
- a set of environment variable values;
- a build architecture; and
- a host architecture,
repeatedly building the source package for the build architecture on
any machine of the host architecture with those versions of the build
dependencies installed and exactly those environment variable values
set will produce bit-for-bit identical binary packages.
It is recommended that packages produce bit-for-bit identical binaries
even if most environment variables and build paths are varied.  It is
intended for this stricter standard to replace the above when it is
easier for packages to meet it.
.. [#]
   This is Debian's precisification of the  reproducible-builds.org
   definition  _.

• Holger Levsen wrote a blog post briefly describing the background and implications of this. To quote him: "we are not 94% done yet, rather more like half done or so. We still need tools and processes to enable anyone to indepently verify that a given binary comes from the sources it is said to be coming, this will involve distributing .buildinfo files and providing user interfaces in APT and elsewhere and probably also systematic rebuilds by us and other parties. And 6% or 7% of the archive is still a lot of packages, eg. in Buster we currently still have 273 unreproducible key packages and for a large part we don't have patches yet so there is still a lot of work ahead."
• There were discussion threads on Hacker News and Reddit.
• Our long-term goal is that Policy mandates that packages "must" be reproducible, but for that we need to show further progress and also reach a consensus on .buildinfo files and much more.

Reproducible work in other projects Bernhard M. Wiedemann's reproducibleopensuse scripts now work on Debian buster on the openSUSE Build Service with the latest versions of osc and obs-build. Toolchain development and fixes #872514 was opened on devscripts by Chris Lamb to add a reproducible-check program to report on the reproducibility status of installed packages. Packages reviewed and fixed, and bugs filed Upstream reports:

• Bernhard M. Wiedemann:
  • qt5/base, SOURCE_DATE_EPOCH support.

Debian reports:

• Adrian Bunk:
  • #872262 filed against jellyfish.
  • #872675 filed against pdp.
  • #872678 filed against gem.
• Chris Lamb:
  • #872453 filed against isa-support.
  • #872459 filed against python-numpy.
  • #872460 filed against gcab, forwarded upstream.
  • #872514 filed against devscripts.
  • #872728 filed against desktop-file-utils.
  • #872729 filed against gtk+2.0, forwarded upstream, found via a reproducibility issue in Tails.
• Federico Brega:
  • #872285 filed against pyqt5.
• Jeremy Bicha:
  • #871963 filed against birdfont.
• Philip Rinn:
  • #872184 filed against quilt.

Debian non-maintainer uploads:

• Mattia Rizzolo:
  • console-data (for bug #799871).

Reviews of unreproducible packages 47 package reviews have been added, 58 have been updated and 39 have been removed in this week, adding to our knowledge about identified issues. 4 issue types have been updated:

• Added nondeterministic_output_generated_by_gcab.
• Added build_path_captured_by_python_numpy_misc_util.
• Added captures_build_path_in_sphinx_attr_links.
• Re-added nondeterminstic_ordering_in_gsettings_glib_enums_xml.

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (59)
• Bastien Roucari s (1)
• James Clarke (1)
• Jeremy Bicha (1)

diffoscope development Development continued in git, including the following contributions:

• Ximin Luo:
  • presenters: html: Don't traverse children whose parents were already limited (Closes: #871413)
  • On a non-GNU system, prefer tools that start with "g" for certain whitelisted commands. (Closes: #871029)
  • Add a --tool-prefix-binutils CLI flag. (Closes: #869868)
• Chris Lamb:
  • Temporarily revert "Bump Standards-Version to 4.0.1" to avoid spurious CI test failures.
  • comparators.xml: Use name attribute over path to avoid leaking comparison full path in output.
  • Code style fixes.

disorderfs development Development continued in git, including the following contributions:

• Chris Lamb:
  • Add simple autopkgtest.

reprotest development Development continued in git, including the following contributions:

• Ximin Luo:
  • Choose an existent HOME for the "control" build. (Closes: #860428)
  • Update debian/changelog with Santiago's changes.
• Santiago Torres:
  • Abstract parts of autopkgtest to support running on non-Debian systems.
  • Add a --host-distro flag to support that too.

tests.reproducible-builds.org Mattia fixed the script which creates the HTML representation of our database scheme to not append .html twice to the filename. Misc. This week's edition was written by Ximin Luo, Chris Lamb and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday 6th and Saturday 12th August 2017:

• There were two talks mainly about Reproducible Builds at DebConf17:
  • Reproducible builds: Status update by Chris Lamb, Holger Levsen, Maria Glukhova, Steven Chamberlain, Vagrant Cascadian, Valerie Young and Ximin Luo.
  • Fun with .buildinfo by Steven Chamberlain.
• Reproducible builds were also mentioned in many other talks including:
  • Using Qubes OS from the POV of a Debian developer by Holger Levsen.
  • Debian for Medical Software by Andr Roth.
  • Let's maintain jenkins.debian.org as a team by Holger Levsen.
  • Signing package contents: why and how by Matthew Garrett.
  • Bits from the DPL by Chris Lamb.
  • Installing Debian by Vagrant Cascadian.
  • in-toto -- Securing supply chains as a whole by Lukas Puehringer.
  • Debian Cloud BoF by Steve McIntyre.
  • Will there be Debian in your next BMW car? by Aigars Mahinovs.
• There has also been substantial activity regarding updating the Debian Policy regarding Reproducible Builds (#844431). This work is still ongoing and will be covered in future issues of this weekly blog.

Notes about reviews of unreproducible packages 13 package reviews have been added, 7 have been updated and 34 have been removed in this week, adding to our knowledge about identified issues. Packages reviewed and fixed, and reproducibility related bugs filed

• Adrian Bunk:
  • #871818 filed against debian-zh-faq.
  • #871907 filed against praat.
• Katsuhiko Nishimra:
  • #871591 filed against llvm-toolchain-snapshot.
• Lucas Nussbaum:
  • #871059 filed against fltk1.3.
  • #871155 filed against brltty.
  • #871159 filed against texlive-extra.
  • #871345 filed against texlive-base.
  • #871349 filed against ispell-uk.
  • #871357 filed against po4a.
• jathan:
  • #870890 filed against apg.

Upstream packages:

• Bernhard M. Wiedemann:
  • enblend (merged), SOURCE_DATE_EPOCH support
  • systemtap
  • splint (dead project)
  • skelcd-openSUSE

Other bugs filed

• During our reproducibility testing, Adrian Bunk filed 48 FTBFS bugs this week.

diffoscope development

• Mattia Rizzolo:
  • debian/copyright: coalesce some file paragraphs and update information.
  • Bump Standards-Version to 4.0.1, no changes needed.

trydiffoscope development

• Chris Lamb:
  • Test --help output in autopkgtests

tests.reproducible-builds.org

• Mattia:
  • Notify the#debian-reproducible-changes IRC channel for unreproducible -> FTBFS transitions.
  • Update squid.conf for all nodes to 5.2.23 (and fixup some).
  • Enable the Munin Squid plugin on the Codethink arm64 nodes as well.
  • Force reconfiguration of Apache and Munin when update_jdn.sh is updated.
• Holger worked on slides for his DebConf17 BoF about migrating to jenkins.debian.org, which affects tests.r-b.o as well.

Misc. This week's edition was written by Chris Lamb & Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday July 30 and Saturday August 5 2017: Media coverage We were mentioned on Late Night Linux Episode 17, around 29:30. Packages reviewed and fixed, and bugs filed Upstream packages:

• Bernhard M. Wiedemann:
  • efl (merged), unique ids based on memory address
  • 389-ds (merged), SOURCE_DATE_EPOCH support.
  • plowshare, SOURCE_DATE_EPOCH support
  • sphinx, file ordering
  • sphinx, SOURCE_DATE_EPOCH support

Debian packages:

• Adrian Bunk:
  • #870212 filed against glib2.0.
  • #870213 filed against pajeng.
  • #870733 filed against openhpi.
  • #870738 filed against gtk2-engines-xfce.
  • #870741 filed against libgda5.
  • #870742 filed against libgnome.
  • #870749 filed against glade.
  • #870821 filed against esys-particle.
• Chris Lamb:
  • #870131 filed against autopep8.
  • #870221 filed against dpkg.
  • #870573 filed against grap.
• Johannes Schauer:
  • #870585 filed against hevea.
• Logan Rosen:
  • #870586 filed against behave.
• Lucas Nussbaum:
  • #871059 filed against fltk1.3.
  • #871155 filed against brltty.
  • #871159 filed against texlive-extra.
• jathan:
  • #870890 filed against apg.

Reviews of unreproducible packages 29 package reviews have been added, 72 have been updated and 151 have been removed in this week, adding to our knowledge about identified issues. 4 issue types have been updated:

• Added timestamps_generated_by_hevea.
• Added timestamps_in_source_generated_by_rcc.
• Updated build_id_differences_only: remove an obsolete example.
• Updated golang_compiler_captures_build_path_in_binary: mark as not deterministic, because the patch fixing it is not yet upstreamed.

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (36)
• Andreas Beckmann (2)
• Daniel Schepler (2)
• Logan Rosen (1)
• Lucas Nussbaum (93)

diffoscope development Version 85 was uploaded to unstable by Mattia Rizzolo. It included contributions from:

• Mattia Rizzolo:
  • Add an explicit Recommends: on the defusedxml python package.
  • Various other code quality tweaks.
• Juliana Oliveira Rodrigues:
  • Fix test_ico_image for ImageMagick identify >= 6.9.8.
  • Use the defusedxml XML library by default in the XML comparator, if it's available. This protects against various XML parser DoS attacks and other security holes, which other Python XML libraries are vulnerable to.
• Ximin Luo:
  • Force a flush when writing output to diff. (Closes: #870049).

as well as previous weeks' contributions, summarised in the changelog. There were also further commits in git, which will be released in a later version:

• Guangyuan Yang:
  • tests/iso9660: support isoinfo's output coming from cdrtools' version instead of genisoimage's
• Mattia Rizzolo:
  • Code quality and test fixes.
• Chris Lamb:
  • Code quality and test fixes.

Misc. This week's edition was written by Ximin Luo, Bernhard M. Wiedemann and Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday July 23 and Saturday July 29 2017:

• Mattia posted an extensive status update from the Debian Reproducible Builds project to the debian-devel-announce mailing list. There were subsequent discussions on Hacker News and Reddit.
• A NASA presentation on "Real system failures" contained an interesting slide pertaining to reproducible builds.
• A 2015 article on Reproducible Builds ("How Debian Is Trying to Shut Down the CIA and Make Software Trustworthy Again") resurfaced in popularity, possibly due to the recent Wikileaks "Vault 7" exposure of malware whose effects can be ameliorated or detected by adopting Reproducible Builds.

Toolchain development and fixes

• Chris Lamb sent an experimental patch to apt to make the output of apt-ftparchive reproducible. Thanks to David Kalnischkies for reworking the result. (#869557)

Packages reviewed and fixed, and bugs filed

• Adrian Bunk:
  • #869578 filed against gdmap.
  • #869580 filed against teg.
  • #869583 filed against gnome-specimen.
  • #869884 filed against chemical-mime-data.
  • #870047 filed against imagemagick.
  • #870068 filed against kde4libs.
• Chris Lamb:
  • #869516 filed against libcdio.
  • A previous pull request against wheel was merged upstream.
• Helmut Grohne:
  • #869584 filed against fontconfig.
  • #869588 filed against libcap2.

Reviews of unreproducible packages 4 package reviews have been added, 2 have been updated and 24 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Aaron M. Ucko (1)
• Adrian Bunk (35)
• Helmut Grohne (4)
• Stefan Tatschner (1)

diffoscope development

• Juliana Oliveira Rodrigues:
  • Refactor XML comparator to use File.recognizes logic
  • Fix test_ico_image for identify >= 6.9.8
  • Fix test_no_android_manifest from test_apk
  • Fix test_android_manifest from test_apk
  • Fix test_xml_expected_diff output encoding to UTF-8
• Ximin Luo:
  • Improve method for detecting APKs
  • Factor common logic from various comparators into File.recognizes

Misc. This week's edition was written by Chris Lamb, Mattia Rizzolo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday July 9 and Saturday July 15 2017: Packages reviewed and fixed, and bugs filed

• Adrian Bunk:
  • #867771 filed against python-trollius.
  • #867773 filed against relatorio.
  • #867781 filed against gcin.
  • #867890 filed against apache-directory-jdbm.
  • #867900 filed against yapsy.
  • #867906 filed against wcsaxes.
• Chris Lamb:
  • #867753 filed against grunt (forwarded upstream).
  • #867848 filed against gconf (forwarded upstream).
  • #868133 filed against grep.
  • #868321 filed against node-marked-man (forwarded-upstream).
• Bernhard M. Wiedemann:
  • chromium
  • chromium
  • kubernetes
  • qtscriptgenerator
  • crash merged
  • ghc-rpm-macros merged
  • samba
  • htmldoc
  • efl
  • webkitgtk3 merged
  • fence-agents
  • fio merged

Reviews of unreproducible packages 13 package reviews have been added, 12 have been updated and 19 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been added:

• build_path_captured_by_valac
• timestamps_in_javascript_generated_by_node_grunt_banner

3 issue types have been updated:

• Add fix for timestamps_embedded_in_manpages_by_node_marked_man toolchain issue.
• Add fix for timestamps_in_javascript_generated_by_node_grunt_banner toolchain issue.
• timestamps_in_javascript_generated_by_node_grunt_banner is actually in src:grunt

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (47)

diffoscope development Version 84 was uploaded to unstable by Mattia Rizzolo. It included contributions already reported from the previous weeks, as well as new ones:

• Ximin Luo:
  • Attempt to fix fsimage test on Jenkins
  • Use a tempdir rather than ./cache for guestfs cache
  • Add more debugging output to test_fsimage

After the release, development continued in git with contributions from:

• Mattia Rizzolo:
  • RequiredToolNotFound.get_package(): just call the new get_package_provider()
  • Add a get_package_provider() function, returning the package name that best matches the system
  • Move from deprecated platform.linux_distribution() to the external package distro

strip-nondeterminism development Versions 0.036-1, 0.037-1 and 0.038-1 were uploaded to unstable by Chris Lamb. They included contributions from:

• Niels Thykier:
  • Add missing use statements in handler modules
  • dh_strip_nondeterminism: Assumes tmpdir() exists
  • File::StripNondeterminism:
    • Apply perltidy
    • Lazy load most handlers
    • Lazy load remaining handlers
• Chris Lamb:
  • Add missing File::Temp imports in JAR and PNG handlers. This appears to have been exposed by lazily-loading handlers in #867982.

reprotest development Development continued in git with contributions from:

• Ximin Luo:
  • presets: use newer flag --no-sign for dpkg-buildpackage
  • Document --diffoscope-args= --exclude-directory-metadata and use it in presets
• Mattia Rizzolo:
  • Bump debhelper compat level to 10.
  • Bump Standards-Version to 4.0.0.

buildinfo.debian.net development

• Chris Lamb:
  • Avoid a race condition between check-and-creation of Buildinfo instances.

tests.reproducible-builds.org

• Mattia Rizzolo:
  • Make database backups quicker to restore by avoiding --column-inserts's pg_dump option.
  • Fixup the deployment scripts after the stretch migration.
  • Fixup Apache redirects that were broken after introducing the buster suite
  • Fixup diffoscope jobs that were not always installing the highest possible version of diffoscope
• Holger Levsen:
  • Add a node health check for a too big jenkins.log.

Misc. This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Mattia Rizzolo, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday June 25 and Saturday July 1 2017: Upcoming and past events Our next IRC meeting is scheduled for July 6th at 17:00 UTC (agenda). Topics to be discussed include an update on our next Summit, a potential NMU campaign, a press release for buster, branding, etc. Toolchain development and fixes

• James McCoy reviewed and merged Ximin Luo's script debpatch into the devscripts Git repository. This is useful for rebasing our patches onto new versions of Debian packages.

Packages fixed and bugs filed

• Adrian Bunk:
  • #866713 filed against debhelper.
• Chris Lamb:
  • #865994 filed against xabacus.
  • #866164 filed against qmidinet.
  • #866169 filed against singularity-container.
  • #866330 filed against cd-hit.

Ximin Luo uploaded dash, sensible-utils and xz-utils to the deferred uploads queue with a delay of 14 days. (We have had patches for these core packages for over a year now and the original maintainers seem inactive so Debian conventions allow for this.) Patches submitted upstream:

• openmpi
• gtk-doc fixed by sorting directory listings
• samba
• TeX
• nedit
• criu
• tvheadend sort
• tvheadend date
• cfengine

Reviews of unreproducible packages 4 package reviews have been added, 4 have been updated and 35 have been removed in this week, adding to our knowledge about identified issues. One issue types has been updated:

• Add upstream URL for random_order_of_pdf_ids_generated_by_latex

One issue type has been added:

• timestamps_in_manpages_added_by_golang_go_flags

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (68)
• Daniel Schepler (1)
• Michael Hudson-Doyle (1)
• Scott Kitterman (6)

diffoscope development

• Daniel Shahaf:
  • Fix markup in the man page synopsis. Thanks to Niels Thykier for the report. (Closes: #866577)
• Mattia Rizzolo:
  • Bump backport version check in debian/rules
• Ximin Luo:
  • Fix a progressbar failure
  • Put the 400MB "fsimage" cache in a more obvious place
  • Fix CI tests under Python 3.6
  • Add a --exclude-directory-metadata option. (Closes: #866241)
  • Raise warning for getfacl. Remove a redundant try-clause
  • Fix recursive indentation of headers
  • Use a loop rather than recursion
  • In html-dir mode, put css/icon in separate files to avoid duplication
  • diffcontrol UI tweaks
  • Split index pages up if they get too big

tests.reproducible-builds.org

• Vagrant Cascadian working on testing Debian:
  • Upgraded the 27 armhf build machines to stretch.
  • Fix mtu check to only display status when eth0 is present.
• Helmut Grohne worked on testing Debian:
  • Limit diffoscope memory usage to 10GB virtual per process. It currently tends to use 50GB virtual, 36GB resident which is bad for everything else sharing the machine. (This is #865660)
• Alexander Couzens working on testing LEDE:
  • Add multiple architectures / targets.
  • Limit LEDE and OpenWrt jobs to only allow one build at the same time using the jenkins build blocker plugin.
  • Move git log -1 > .html to node document environment().
  • Update TODOs.
• Mattia Rizzolo working on testing Debian:
  • Updated the maintenance script for postgresql-9.6.
  • Restored the postgresql database from backup after Holger accidently purged it.
• Holger Levsen working on:
  • Merging all the above commits.
  • Added a check for (known) Jenkins zombie jobs and report them. (This is an long known problem with jenkins; deleted jobs sometimes come back )
  • Upgraded the remaining amd64 nodes to stretch.
  • Accidentially purged postgres-9.4 from jenkins, so we could test our backups
  • Updated our stretch upgrade TODOs.

Misc. This week's edition was written by Chris Lamb, Ximin Luo, Holger Levsen, Bernhard Wiedemann, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday June 18 and Saturday June 24 2017: Upcoming and Past events Our next IRC meeting is scheduled for the 6th of July at 17:00 UTC with this agenda currently:

1. Introductions
2. Reproducible Builds Summit update
3. NMU campaign for buster
4. Press release: Debian is doing Reproducible Builds for Buster
5. Reproducible Builds Branding & Logo
6. should we become an SPI member
7. Next meeting
8. Any other business

On June 19th, Chris Lamb presented at LinuxCon China 2017 on Reproducible Builds. On June 23rd, Vagrant Cascadian held a Reproducible Builds question and answer session at Open Source Bridge. Reproducible work in other projects LEDE: firmware-utils and mtd-utils/mkfs.jffs2 now honor SOURCE_DATE_EPOCH. Toolchain development and fixes There was discussion on #782654 about packaging bazel for Debian. Dan Kegel wrote a patch to use ar determinitiscally for Homebrew, a package manager for MacOS. Dan Kegel worked on using SOURCE_DATE_EPOCH and other reproduciblity fixes in fpm, a multi plattform package builder. The Fedora Haskell team disabled parallel builds to achieve reproducible builds. Bernhard M. Wiedemann submitted many patches upstream:

• Sorting file lists:
  • python setuptools
  • cpython tarfile/zipfile
  • lyx
  • dpdk
  • perl Alien::SDL
  • gtk-doc, no fix yet
• Use date from SOURCE_DATE_EPOCH:
  • uwsgi
  • recode
  • perl Glib

Packages fixed and bugs filed Patches submitted upstream:

• Bernhard M. Wiedemann
  • samba, sorting file lists
  • open-mpi, merged, sorting file lists
  • virtualbox, incomplete fix, use gzip -n
  • xen/etherboot, xen/mini-os, use gzip -n
  • nautilus-dropbox, drop date
  • samba, use date from SOURCE_DATE_EPOCH.

Other patches filed in Debian:

• Adrian Bunk:
  • #865577 filed against cloud-init.
  • #865813 filed against jumpnbump.
• Chris Lamb:
  • #865554 filed against janus.
  • #865623 filed against cracklib2.
  • #865751 filed against nfstrace.
  • #865753 filed against tigervnc.

Reviews of unreproducible packages 573 package reviews have been added, 154 have been updated and 9 have been removed in this week, adding to our knowledge about identified issues. 1 issue type has been updated:

• random_order_of_pdf_ids_generated_by_latex, added the name of the function that possibly causes this issue.

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (98)

diffoscope development Version 83 was uploaded to unstable by Chris Lamb. It also moved the previous changes from experimental (to where they were uploaded) to unstable. It included contributions from previous weeks. You can read about these changes in our previous weeks' posts, or view the changelog directly (raw form). We plan to maintain a backport of this and future versions in stretch-backports. Ximin Luo also worked on better html-dir output for very very large diffs such as those for GCC. So far, this includes unreleased work on a PartialString data structure which will form a core part of a new and more intelligent recursive display algorithm. strip-nondeterminism development Versions 0.035-1 was uploaded to unstable from experimental by Chris Lamb. It included contributions from:

• Bernhard M. Wiedemann
  • Add CPIO handler and test case.
• Chris Lamb
  • Packaging improvements.

Later in the week Mattia Rizzolo uploaded 0.035-2 with some improvements to the autopkgtest and to the general packaging. We currently don't plan to maintain a backport in stretch-backports like we did for jessie-backports. Please speak up if you think otherwise. reproducible-website development

• Chris Lamb:
  • Add OpenEmbedded to projects page after a discussion at LinuxCon China.
  • Update some metadata for existing talks.
  • Add 13 missing talks.

tests.reproducible-builds.org

• Alexander 'lynxis' Couzens
  • LEDE: do a quick sha256sum before calling diffoscope. The LEDE build consists of 1000 packages, using diffoscope to detect whether two packages are identical takes 3 seconds in average, while calling sha256sum on those small packages takes less than a second, so this reduces the runtime from 3h to 2h (roughly). For Debian package builds this is neglectable, as each build takes several minutes anyway, thus adding 3 seconds to each build doesn't matter much.
  • LEDE/OpenWrt: move toolchain.html creation to remote node, as this is were the toolchain is build.
  • LEDE: remove debugging output for images.
  • LEDE: fixup HTML creation for toolchain, build path, downloaded software and GIT commit used.
• Mattia 'mapreri' Rizzolo:
  • Debian: introduce Buster.
  • Debian: explain how to migrate from squid3 (in jessie) to squid (in stretch).
• Holger 'h01ger' Levsen:
  • Debian:
    • Add jenkins jobs to create schroots and configure pbuilder for Buster.
    • Add Buster to README/about jenkins.d.n.
    • Teach jessie and ubuntu 16.04 systems how to debootstrap Buster.
    • Only update indexes and pkg_sets every 30min as the jobs almost run for 15 min now that we test four suites (compared to three before).
    • Create HTML dashboard, live status and dd-list pages less often.
    • (Almost) stop scheduling old packages in stretch, new versions will still be scheduled and tested as usual.
    • Increase scheduling limits, especially for untested, new and depwait packages.
    • Replace Stretch with Buster in the repository comparison page.
    • Only keep build_service logs for a day, not three.
    • Add check for hanging mounts to node health checks.
    • Add check for haveged to node health checks.
    • Disable ntp.service on hosts running in the future, needed on stretch.
    • Install amd64 kernels on all i386 systems. There is a performance issue with i386 kernels, for which a bug should be filed. Installing the amd64 kernel is a sufficient workaround, but it breaks our 32/64 bit kernel variation on i386.
  • LEDE, OpenWrt: Fix up links and split TODO list.
  • Upgrade i386 systems (used for Debian) and pb3+4-amd64 (used for coreboot, LEDE, OpenWrt, NetBSD, Fedora and Arch Linux tests) to Stretch
  • jenkins: use java 8 as required by jenkins >= 2.60.1

Misc. This week's edition was written by Ximin Luo, Holger Levsen, Bernhard M. Wiedemann, Mattia Rizzolo, Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.